How would you evaluate whether a change to your recommendation retrieval pipeline actually improved results, before rolling it out to all users?

The evaluation has two stages: offline and online. Offline, you compute relevance metrics on a held-out dataset using a ground-truth definition of relevance. The standard metrics are nDCG (normalized discounted cumulative gain) and MRR (mean reciprocal rank) for ranking quality, and recall@k for coverage. For recommendation, the ground truth is typically derived from user interactions - items the user engaged with in the future are treated as relevant, and the metrics measure how well the new pipeline ranks those items. The offline evaluation is fast, cheap, and lets you compare many variants, but it has a well-known limitation: it measures ranking quality against a fixed ground truth, which may not reflect the actual user experience or the business metrics that matter. A pipeline can improve nDCG while hurting click-through, so offline results are necessary but not sufficient. The online stage is a staged A/B test: the new pipeline is deployed to a small fraction of traffic, and the metrics that matter (click-through, conversion, retention, revenue) are compared with the control group. The staged approach limits the blast radius of a bad change and gives a statistically sound comparison.

The mechanism that makes this work is that the two stages answer different questions. Offline evaluation answers 'does the new pipeline rank relevant items higher on the historical data?' Online evaluation answers 'does the new pipeline produce better outcomes for real users?' The offline stage filters out clearly bad changes and lets you iterate quickly; the online stage validates the ones that pass. The offline ground truth is the key design decision: if it is derived from future interactions, it captures the user's actual preferences; if it is derived from a hand-labeled set, it captures expert judgment but may not reflect the user population. The online experiment must control for confounding - the treatment and control groups must be sampled from the same population, the time period must be the same, and the metrics must be pre-registered to avoid p-hacking. The experiment should run long enough to reach statistical significance, which depends on the effect size and the traffic volume. A common practice is to run for at least one week to capture weekly patterns, and to use a sequential test or a fixed-horizon test with a pre-computed sample size.

The trade-off is between speed and confidence. Offline evaluation is fast but imperfect; online evaluation is definitive but slow and risky. The right balance is to use offline to filter and online to validate, with a staged rollout that limits the risk. The common mistake is to ship on offline metrics alone, which can lead to a change that improves nDCG but hurts revenue. The second mistake is to run an A/B test without enough traffic or for too short a time, producing a result that is not statistically significant or that is confounded by a seasonal effect. The third mistake is to not pre-register the metrics, so the analysis becomes a search for a metric that happens to improve. The fourth mistake is to ignore guardrail metrics, so a change that improves the target metric but increases latency or error rates slips through. The fifth mistake is to evaluate only the retrieval pipeline in isolation, without considering the reranking and generation stages downstream - a change in retrieval can have non-obvious effects on the final answer quality. Version note: the evaluation methodology is version-independent, but the Qdrant features you are evaluating (quantization, hybrid search, reranking) have evolved across releases, so re-run the offline evaluation after upgrading Qdrant.

Version-dependent: the evaluation methodology is version-independent, but the Qdrant features being evaluated (quantization, hybrid search, reranking) have evolved across releases. If the new pipeline uses a feature that is version-specific, benchmark it on the version you plan to deploy. The offline evaluation uses the same Qdrant API as production, so the query_points API shape (qdrant-client 1.10+) applies.